Back Door Manipulation

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 327 of xUnit Test Patterns for the latest information.
Also known as: Layer-Crossing Test

How can we verify logic independently when we cannot use a round trip test?

We set up the test fixture or verify the outcome by going through a back door (such as direct database access.)

Sketch Back Door Manipulation embedded from Back Door Manipulation.gif

Every test requires a starting point (the test fixture) and an expected finishing point (the expected results). The "normal" approach is to set up the fixture and verify the outcome by using the API of the system under test (SUT) itself. In some circumstances this is either not possible or not desirable.

In some situations we can use Back Door Manipulation to set up the fixture and/or verify the SUT's state.

How It Works

The state of the SUT comes in many flavors. It can be stored in memory, on disk as files, in a database or in other applications with which the SUT interacts. Whatever form it takes, the pre-condition of a test typically requires that the state of the SUT not only is known but is a specific state. Likewise, at the end of the test we often want to do State Verification (page X) of the SUT's state.

If we have access to the state of the SUT from outside the SUT, the test can set up the pre-test state of the SUT by bypassing the normal API of the SUT and interacting directly with whatever is holding that state via a "back door". When exercising of the SUT has been completed, the test can access the post-test state of the SUT via a "back door" to compare it with expected outcome. For customer tests, the "back door" is most commonly a test database but it could be some other component on which the SUT depends including a Registry[PEAA] object or even the file system. For unit tests, the back door is some other class or object or an alternate interface of the SUT (or a Test-Specific Subclass (page X)) that exposes the state in a way "normal" clients wouldn't use. We can also replace a depended-on component (DOC) with a suitably configured Test Double (page X) instead of using the real thing if that makes the job easier.

When To Use It

We might choose to use Back Door Manipulation for several reasons which I'll go into more detail shortly. A prerequisite for using Back Door Manipulation is that there is some sort of "back door" to the state of the system. The main drawback of Back Door Manipulation is that our tests (or the Test Utility Methods (page X) they call) are now much more closely coupled to the design decisions we make about how the state of the SUT is represented. If those decisions need to be changed, we may encounter Fragile Tests (page X). We need to decide whether this price is acceptable on a case by case basis. The impact of the close coupling can be greatly reduced by encapsulating all Back Door Manipulation in Test Utility Methods.

Using Back Door Manipulation can also lead to Obscure Tests (page X) by making less obvious the relationship of the test outcome to the test fixture. This can be avoided by including the test data being passed to the Back Door Manipulation mechanism within the Testcase Class (page X) or mitigated by using Finder Methods (see Test Utility Method) to refer to the objects in the fixture via intent-revealing names.

A common application of Back Door Manipulation is when testing basic CRUD (Create, Read, Update and Delete) operations on the SUT's state. The requirement we want to verify is that the information is persisted and can be recovered in the same form. It is difficult to write round trip tests for "Read" without also testing "Create"; likewise, it is difficult to test "Update" or "Delete" without testing both "Create" and "Read". We can certainly test them in these groups using round trip tests but this won't detect certain types of systemic problems such putting information into the wrong database column. One solution to use layer-crossing tests that use Back Door Manipulation to set up or verify the contents of the database directly. For a "Read" test, the test sets up the contents of the database using Back Door Setup and then ask the SUT to read the data. For a "Write" test, the test asks the system to write certain objects and then uses Back Door Verification on the contents of the database.

Variation: Back Door Setup

One reason for doing Back Door Manipulation is to make tests run faster. If a system does a lot of processing before putting data into its data store, the time it takes for a test to set up the fixture via the SUT's API could be quite significant. One way to make the tests run faster is to determine what those data stores should look like and then create a means to set them up via the back door rather than through the API. A problem this introduces is that because Back Door Setup bypasses enforcement of the object creation business rules, we may find ourselves creating fixtures that are not realistic and possibly even invalid. This problem may only creep in over time as the business rules get modified in response to changing business needs. On the other hand, it may allow us to set up test scenarios that the SUT will not let us set up through its API.

When we share a database between our SUT and another application, we need to verify that we are using the database correctly as well as verifying that we can handle all possible data configurations the other applications might create. Back Door Setup is a good way to set up these configurations and may be the only way if our SUT either doesn't write those tables or only writes specific (and valid) data configurations. Back Door Setup lets us create those "impossible" configurations easily so we can verify how our SUT behaves in these situations.

Variation: Back Door Verification

Back Door Verification involves sneaking in to do State Verification of the SUT's post-exercise state via a "back door" and is mostly applicable to customer tests (or functional tests as they are sometimes called.) The back door is typically an alternative way to examine the objects in the database usually through a standard API such as SQL or via data exports that can then be examined with a file comparison utility program.

One reason for using Back Door Manipulation is to make tests run faster. If the only way to get at the SUT's state is to invoke an expensive operation (such as a complex report) or an operation that further modifies the SUT's state, we may be better off using Back Door Manipulation.

Another reason for doing Back Door Manipulation is that other systems expect the SUT to store it's state in a specific way that they access directly. This is an example of an indirect output. In this situation, standard round trip tests are insufficient to prove that the SUT's behavior is correct because they won't detect a systematic problem if the write and read make the same mistake such as putting information into the wrong database column. The solution is a layer-crossing test that looks at the contents of the database directly to verify that the information was stored correctly. For a "write" test, the test asks the system to write certain objects and then inspects the contents of the database via the back door.

Variation: Back Door TearDown

We can also use Back Door Manipulation to tear down a Fresh Fixture (page X) that is stored in a test database. This is especially beneficial if we can use bulk database commands to wipe clean whole tables as in Table Truncation Teardown (page X) or Transaction Rollback Teardown (page X).

Implementation Notes

How we implement Back Door Manipulation depends on where the fixture lives and how easy it is to access the state of the SUT. It also depends on why we are doing Back Door Manipulation. I'm listing the most common implementations here but feel free to use your imagination and come up with other ways.

Variation: Database Population Script

When the SUT we are exercising stores its state in a database that it accesses as it runs, the easiest way to do Back Door Manipulation is to load data directly into that database before invoking the SUT. This is most commonly required when writing customer tests but may also be required for unit tests if the classes we are testing interact directly with the database. To do this we determine what the preconditions of the test are and from that what data the test requires for its fixture. We then define a database script that inserts the corresponding records directly into the database bypassing the SUT logic. We use this Database Population Script whenever we want to set up the test fixture. That depends on which test fixture strategy we have chosen. (See Test Automation Strategy for more on that topic.)

When deciding to use a Database Population Script, we will need to maintain both the Database Population Script and the files it takes as input whenever we modify either the structure of the SUT's data stores or the semantics of the data in them. This can increase the maintenance cost of the tests.

Variation: Data Loader

A Data Loader is a special program that loads data into the SUT's data store. The main difference from a Database Population Script is that it is written in a programming language rather than a database language. This gives us a bit more flexibility and allows us to use it even when the system state is stored in places other than a relational database.

If the data store is external to the SUT, such as in a relational database, the Data Loader can be "just another application" that writes to that data store. It would use the database in much the same way as the SUT but it would get its inputs from a file rather than from wherever the SUT normally gets its inputs (e.g. other "upstream" programs.) When we are using an "Object Relational Mapping" (ORM) tool to access the database from our SUT, a simple way to build the Data Loader is to use the same domain objects and mappings in our Data Loader. We just create the desired objects in memory and commit the ORM's unit of work to save them into the database.

If the SUT stores data in internal data structures (e.g. in memory), the Data Loader may need to be an interface provided by the SUT itself. What differentiates it from the normal functionality provided by the SUT is:

It is only used by the tests.
It reads the data from a file rather than wherever the SUT normally gets the data.
It bypasses a lot of the "edit checks" (input validation) normally done by the SUT.

The input files may be simple flat files containing comma or tab delimited text, or they could be structured using XML. DbUnit is an extension of JUnit that implements Data Loader for fixture set up.

Variation: Database Extraction Script

When the SUT we are exercising stores its state in a database that it accesses as it runs, we can take advantage of this to do Back Door Verification. We simply use a database script to extract data from the test database and verify it contains the right data either by comparing it to previously prepared "extract" files or by ensuring the right number of records are returned by very specific queries.

Variation: Data Retriever

A Data Retriever is the analog of a Data Loader for the purpose of retrieving state from the SUT when doing Back Door Verification. Like our trusty dog, it "fetches" the data so that we can compare it with our expected results within our tests. DbUnit is an extension of JUnit that implements Data Retriever to support result verification.

Variation: Test Doubles

So far, all the implementation techniques have involved interacting with a DOC of the SUT to set up or tear down the fixture or to verify the expected outcome. Probably the most common form of Back Door Manipulation is to replace the DOC with a Test Double. One option is to use a Fake Object (page X) that we have preloaded with some data as though the SUT had already been interacting with it; this is a way to avoid using the SUT to set up the SUT's state. The other option is to use some kind of Configurable Test Double (page X) such as a Mock Object (page X) or a Test Stub (page X). Either ways, we can completely avoid Obscure Test by making the state of the Test Double visible within the Test Method (page X).

When we want to do Behavior Verification (page X) of the calls made by the SUT to one or more DOC's, we can use a layer-crossing test that replaces the DOC with a Test Spy (page X) or a Mock Object. When we want to verify that the SUT behaves a specific way when it receives indirect inputs from a DOC (or when in some specific external state), we can replace the DOC with a Test Stub.

Motivating Example

The following round trip test verifies the basic functionality of removing a flight by interacting with the SUT only via the front door. But it does not verify the indirect outputs of the SUT, namely, that the SUT is expected to call a logger to log each time a flight is removed along with day/time and user id of the requester. In many systems, this would be an example of "layer-crossing behavior" because the logger would be part of a generic infrastructure layer while the SUT itself is an application-specific behavior.

   public void testRemoveFlight() throws Exception {
      // setup
      FlightDto expectedFlightDto = createARegisteredFlight();
      FlightManagementFacade facade = new FlightManagementFacadeImpl();
      // exercise
      facade.removeFlight(expectedFlightDto.getFlightNumber());
      // verify
      assertFalse("flight should not exist after being removed",
                  facade.flightExists( expectedFlightDto.getFlightNumber()));
   }
Example UntestedRequirementTest embedded from java/com/clrstream/ex8/test/FlightManagementFacadeTest.java

Refactoring Notes

We can convert this test to use Back Door Verification by adding result verification code to access and verify the logger's state. We can do this either by reading that state from the logger's database or by replacing the logger with a Test Spy that saves the state for easy access by the tests.

Example: Back Door Result Verification Using a Test Spy

Here's the same test converted to use a Test Spy to access the post-test state of the logger.

   public void testRemoveFlightLogging_recordingTestStub() throws Exception {
      // fixture setup
      FlightDto expectedFlightDto = createAnUnregFlight();
      FlightManagementFacade facade = new FlightManagementFacadeImpl();
      //    Test Double setup
      AuditLogSpy logSpy = new AuditLogSpy();
      facade.setAuditLog(logSpy);
      // exercise
      facade.removeFlight(expectedFlightDto.getFlightNumber());
      // verify
      assertEquals("number of calls", 1, logSpy.getNumberOfCalls());
      assertEquals("action code", Helper.REMOVE_FLIGHT_ACTION_CODE,
                   logSpy.getActionCode());
      assertEquals("date", helper.getTodaysDateWithoutTime(), logSpy.getDate());
      assertEquals("user", Helper.TEST_USER_NAME, logSpy.getUser());
      assertEquals("detail", expectedFlightDto.getFlightNumber(),
                   logSpy.getDetail());
   }
Example ProceduralBehaviorVerification embedded from java/com/clrstream/ex8/test/FlightManagementFacadeTestSolution.java

This would be the better way to do this if there was so much data in the logger's database that it wasn't practical to verify the new entries using Delta Assertions (page X).

Example: Back Door Fixture Setup

This example shows how we can set up a fixture using the database as a back door to the SUT. The test inserts a record into the EmailSubscription table and then asks the SUT to find it. It then asserts on various fields of the object returned by the SUT to verify the record was read correctly.

   static final String     TABLE_NAME = "EmailSubscription";
   static final BigDecimal RECORD_ID  = new BigDecimal("111");
  
   static final String LOGIN_ID = "Bob";
   static final String EMAIL_ID = "bob@foo.com";
  
   public void setUp() throws Exception {
      String xmlString = "<?xml version='1.0' encoding='UTF-8'?>" +
            "<dataset>" +
            "    <" + TABLE_NAME +
            "        EmailSubscriptionId='" + RECORD_ID +  "'" +
            "        UserLoginId='" + LOGIN_ID + "'" +
            "        EmailAddress='" + EMAIL_ID + "'" +
            "        RecordVersionNum='62' " +
            "        CreateByUserId='MappingTest' " +
            "        CreateDateTime='2004-03-01 00:00:00.0'  " +
            "        LastModByUserId='MappingTest' " +
            "        LastModDateTime='2004-03-01 00:00:00.0'/>" +
            "</dataset>";
      insertRowsIntoDatabase(xmlString);
   }
  
   public void testRead_Login() throws Exception {
      // exercise:
      EmailSubscription subs = EmailSubscription.findInstanceWithId(RECORD_ID);
      // verify
      assertNotNull("Email Subscription", subs);
      assertEquals("User Name", LOGIN_ID, subs.getUserName());
   }
  
   public void testRead_Email() throws Exception {
      // exercise:
      EmailSubscription subs = EmailSubscription.findInstanceWithId(RECORD_ID);
      // verify
      assertNotNull("Email Subscription", subs);
      assertEquals("Email Address", EMAIL_ID,
                   subs.getEmailAddress());
   }
Example DbUnitReadTest embedded from java/com/xunitpatterns/dbunitExample/EmailSubscriptionReadOnlyTest.java

I have built the XML document used to populate the database within the Testcase Class to avoid the Mystery Guest (see Obscure Test) that would have been caused by using an external file for loading the database (see Inline Resource (page X) refactoring to see how to do this.) To make the test clearer I am calling intent-revealing methods that hide the details of how we use DbUnit to load the database and clean it out at the end of the test using Table Truncation Teardown. Here are the bodies of the Test Utility Methods I am using:

   private void insertRowsIntoDatabase(String xmlString) throws Exception {
      IDataSet dataSet = new FlatXmlDataSet(new StringReader(xmlString));
      DatabaseOperation.CLEAN_INSERT.execute( getDbConnection(), dataSet);
   }
  
   public void tearDown() throws Exception{
      emptyTable(TABLE_NAME);
   }
  
   public void emptyTable(String tableName) throws Exception {
      IDataSet dataSet = new DefaultDataSet(new DefaultTable(tableName));
      DatabaseOperation.DELETE_ALL.execute(getDbConnection(), dataSet);
   }
Example DbUnitReadTestUtilities embedded from java/com/xunitpatterns/dbunitExample/EmailSubscriptionReadOnlyTest.java

Of course, the implementations of these methods are specific to DbUnit; we would need to change them if we were using some other member of the xUnit family.

Some other observations on these tests: To avoid an Eager Test (see Assertion Roulette on page X), I have put the assertion on each field into a separate test. This could result in Slow Tests (page X) because these tests do interact with a database. I could use Lazy Initialization[SBPP] or SuiteFixture Setup (page X) to avoid setting it up more than once as long as the resulting Shared Fixture (page X) was not modified by any of the tests. (I chose not to further complicate this example by doing this.)